Hand tracker for device with display

ABSTRACT

An apparatus for remote hand control of a computer device, the apparatus comprising: a hand tracker, configured to track movement of a hand of a user of the computer device, an image presenter, associated with the hand tracker, configured to present a first image representative of the hand and a second image to the user, and to move the first image in concert with the tracked movement, and a computer controller, associated with the image presenter and configured to control the computer device in accordance with an interaction between the first image and the second image, thereby allowing the user to control the computer device, by moving the hand for bringing the first image and the second image into the interaction.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.15/057,382, filed Mar. 1, 2016, which is a Continuation of U.S. patentapplication Ser. No. 13/497,589, filed Mar. 22, 2012, now U.S. Pat. No.9,507,411, issued Nov. 29, 2016, which is the National Stage ofInternational Patent Application of PCT/IB2010/054228, filed Sep. 19,2010, which claims the benefit of U.S. Provisional Patent ApplicationSer. No. 61/244,473, filed Sep. 22, 2009, each of which is herebyincorporated by reference in its entirety.

FIELD AND BACKGROUND OF THE INVENTION

The present invention relates to control of computer devices and, moreparticularly, but not exclusively to a system and method for remotecontrol of computer devices.

Currently, there is a variety of methods used for remote control ofcomputer devices.

Some of the currently used methods enable a computer user to control acomputer device with a predefined gesture, using one or more body parts,such as the user's arms or legs.

With the currently used methods, there is defined a discrete set ofgestures. Once a user's body part seems to align in a predefinedposition, a gesture is detected. Consequently, the computer devicecarries out a predefined function.

With current methods, each gesture, once detected, activates computeractions which are predefined for the specific gesture. The currentmethods typically include an initial set-up stage. In the set-up stage,there is defined a discrete set of gestures and computer functions foreach specific gesture in the set.

The gesture may be detected through a variety of currently used methods.

For example, some of the current methods include regular image analysisfrom a digital video stream. The video images are analyzed to detect thedominant body part's location and position. If the body parts arealigned in a predefined position, the predefined function is carried outby the computer device.

With the regular image analysis methods, the detection of the body partis carried out by analysis of each pixel in the video image. The pixelis analyzed by a comparison made between the pixel's color values andvalues of other pixels in proximity to the pixel. That is to say thatthe regular image analysis methods depend on a significant difference incolor between the body part and background objects.

Other currently used methods are based on a computationally heavycalculation of a three dimensional depth map.

A depth map is an image that holds in each pixel the distance to thepart of the object located in front of a depth camera.

With the three dimensional depth map, extraction of the location of theuser's hands may prove relatively easy, since the hands are typicallylocated in front of the rest of the user's body. Consequently, parts ofthe three dimensional map that are located in a distance exceeding acertain distance, may be discarded.

The depth map may be calculated using a variety of methods. For example,in stereovision methods, two or more cameras are used to capture imagesof the user's body. Images of an object captured from the cameras arecompared and analyzed, to produce three dimensional data of depthlocation of each point on the surface of the user's body, thus yieldingthe depth map.

In shape from shading methods, the user's body is lit from severaldirections.

The shades of the body are compared and analyzed, to generate threedimensional data of the location of each point on the surface of theuser's body, thus yielding the depth map.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention there is providedan apparatus for remote hand control of a computer device, the apparatuscomprising: a hand tracker, configured to track movement of a hand of auser of the computer device, an image presenter, associated with thehand tracker, configured to present a first image representative of thehand and a second image to the user, and to move the first image inconcert with the tracked movement, and a computer controller, associatedwith the image presenter and configured to control the computer devicein accordance with an interaction between the first image and the secondimage, thereby allowing the user to control the computer device, bymoving the hand for bringing the first image and the second image intothe interaction.

According to a second aspect of the present invention there is provideda computer implemented method for remote hand control of a computerdevice, the method comprising steps the computer device is programmed toperform, the steps comprising: tracking movement of a hand of a user ofthe computer device, presenting a first image representative of the handand a second image to the user, and moving the first image in concertwith the tracked movement, and controlling the computer device inaccordance with an interaction between the first image and the secondimage, thereby allowing the user to control the computer device, bymoving the hand for bringing the first image and the second image intothe interaction.

According to a third aspect of the present invention there is provided acomputer readable medium storing computer executable instructions forperforming steps of remote hand control of a computer device, the stepscomprising: tracking movement of a hand of a user of the computerdevice, presenting a first image representative of the hand and a secondimage to the user, and moving the first image in concert with thetracked movement, and controlling the computer device in accordance withan interaction between the first image and the second image, therebyallowing the user to control the computer device, by moving the hand forbringing the first image and the second image into the interaction.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. The materials, methods, andexamples provided herein are illustrative only and not intended to belimiting.

Implementation of the method and system of the present inventioninvolves performing or completing certain selected tasks or stepsmanually, automatically, or a combination thereof. Moreover, accordingto actual instrumentation and equipment of preferred embodiments of themethod and system of the present invention, several selected steps couldbe implemented by hardware or by software on any operating system of anyfirmware or a combination thereof.

For example, as hardware, selected steps of the invention could beimplemented as a chip or a circuit. As software, selected steps of theinvention could be implemented as a plurality of software instructionsbeing executed by a computer using any suitable operating system. In anycase, selected steps of the method and system of the invention could bedescribed as being performed by a data processor, such as a computingplatform for executing a plurality of instructions.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings. With specific reference now tothe drawings in detail, it is stressed that the particulars shown are byway of example and for purposes of illustrative discussion of thepreferred embodiments of the present invention only, and are presentedin order to provide what is believed to be the most useful and readilyunderstood description of the principles and conceptual aspects of theinvention. The description taken with the drawings making apparent tothose skilled in the art how the several forms of the invention may beembodied in practice.

In the drawings:

FIG. 1 is a block diagram schematically illustrating an apparatus forremote hand control of a computer device, according to an exemplaryembodiment of the present invention.

FIG. 2 is a flowchart illustrating a method for remote hand control of acomputer device, according to an exemplary embodiment of the presentinvention.

FIG. 3 is a block diagram schematically illustrating a computer readablemedium storing computer executable instructions for performing steps ofremote hand control of a computer device, according to an exemplaryembodiment of the present invention.

FIG. 4 schematically illustrates a hand projected with a light pattern,according to an exemplary embodiment of the present invention.

FIG. 5 schematically illustrates a hand projected with a light patternhaving an intensity notation, according to an exemplary embodiment ofthe present invention.

FIG. 6 is a block diagram schematically illustrating remote hand controlof a computer device, using an image of a computer keyboard, accordingto an exemplary embodiment of the present invention.

FIG. 7 is a block diagram schematically illustrating a first gesture forremote hand control of a computer device, according to an exemplaryembodiment of the present invention.

FIG. 8 is a block diagram schematically illustrating a second gesturefor remote hand control of a computer device, according to an exemplaryembodiment of the present invention.

FIG. 9 is a block diagram schematically illustrating a third gesture forremote hand control of a computer device, according to an exemplaryembodiment of the present invention.

FIG. 10 is a block diagram schematically illustrating a fourth gesturefor remote hand control of a computer device, according to an exemplaryembodiment of the present invention.

FIG. 11 is a block diagram schematically illustrating a system forremote hand control of a computer device, according to an exemplaryembodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present embodiments comprise an apparatus and a method, for remotehand control of a computer device.

According to an exemplary embodiment of the present invention, movementof a hand of a user of a computer device is continuously tracked.

The user is presented a first image representative of the hand and asecond image.

The first image may be a full and accurately animated image of theuser's hand, presented on a screen. Alternatively, the first imagerather consists of five cursors, which represent the locations of thehand fingertips. The first image may also have any other visualappearance which gives the user information about the location of digits(i.e. fingers, thumb, or both) in relation to the second image, asdescribed in further detail hereinbelow.

As the user moves his hand (or only his fingers or thumb), the firstimage is moved in concert with the tracked movement. That is to say thatthe movement of the first image is correlated with the movement of theuser's hand. If the user moves his hand to the left, the first imagemoves to the left. If the user flexes one of his fingers, the firstimage also flexes the finger (or moves the cursor representative oflocation of the finger's tip accordingly), etc.

Optionally, the second image is a graphical user interface (GUI) of acomputer application such as a Microsoft™ Excel spreadsheet with buttonsand menu options, a web page with hyperlinks, buttons, etc.

The computer device is controlled according to interaction between thefirst image and the second image, as if the first image is the user'sown hand, and the second image is a GUI presented on a touch screenwhich allows the user to interact with the GUI's menu, buttons, etc., bytouching the touch screen with his own fingers.

That is to say that the user is allowed to control the computer device,by moving the user's hand or fingers, for bringing the first image andthe second image into the interaction.

Optionally, the second image is rather an image representative of acomputer input device (say a computer keyboard, a computer mouse, ajoystick, etc., as known in the art).

The computer device is controlled according to interaction between thefirst image and the second image, as if the hand in the image is theuser's own hand, and the second image is a real computer input deviceconnected to the computer device like any standard peripheral computerdevice (say a keyboard, a joystick, etc). The user is thus allowed tocontrol the computer device, by moving the user's hand, for bringing thefirst image and the second image into the interaction.

Consequently, the remote control of the computer device, by movement ofthe user's hand, is a based on a continuous user interface, rather thanon an interface limited to a predefined set of specific gestures (thoughspecific gestures may also be defined).

In one example, if the user moves his hand, the first image moves overan image of a computer keyboard, and if the user moves his fingers in atyping movement, the computer device responds as if the user types usinga real keyboard, as described in further detail hereinbelow.

The principles and operation of an apparatus and a method, according toexemplary embodiments of the present invention may be better understoodwith reference to the drawings and accompanying description.

Before explaining at least one embodiment of the invention in detail, itis to be understood that the invention is not limited in its applicationto the details of construction and the arrangement of the components setforth in the following description or illustrated in the drawings.

The invention is capable of other embodiments or of being practiced orcarried out in various ways. Also, it is to be understood that thephraseology and terminology employed herein is for the purpose ofdescription and should not be regarded as limiting.

Reference is now made to FIG. 1, which is a block diagram schematicallyillustrating an apparatus for remote hand control of a computer device,according to an exemplary embodiment of the present invention.

Apparatus 1000 may be implemented on the computer device, be thecomputer device a desktop computer, a laptop computer, a cellular phone,etc. The apparatus 1000 may also be implemented on a device connectableto the computer device, say on a unit with a computer processor incommunication with the computer device. Further, the apparatus 1000 maybe implemented as software, as hardware, or as a combination of softwareand hardware.

The apparatus 1000 includes a hand tracker 110.

The hand tracker 110 tracks movement of a hand of a user of the computerdevice, say using images of the hand projected with light structured ina pattern useful for tracking fine movement of fingers and thumbs, asdescribed in further detail hereinbelow.

Optionally, the hand tracker 110 is remote from the computer device, asdescribed in further detail hereinbelow.

The apparatus 1000 further includes an image presenter 120, incommunication with the hand tracker 110.

The image presenter 120 presents a first image representative of thehand and a second image.

Optionally, the second image is a graphical user interface (GUI) of acomputer application such as a Microsoft™ Excel spreadsheet with buttonsand menu options, a web page with hyperlinks, buttons, etc.

Optionally, the second image is rather representative of a computerinput device say a computer mouse or a computer keyboard, as describedin further detail hereinbelow.

Optionally, the second image is another graphical object, a picture,etc. The first image may be a full and accurately animated image of theuser's hand, presented on the screen. Alternatively, the first imagerather consists of five cursors, which represent the locations of thehand's fingertips. The first image may also have any other visualappearance which gives the user information about the location of digits(i.e. fingers, thumb, or both) in relation to the second image.

The image presenter 120 further moves the first image in concert withthe user hand movement tracked by the hand tracker 110, as described infurther detail hereinbelow.

Apparatus 1000 further includes a computer controller 130, incommunication with the image presenter 120.

Optionally, the computer controller 130 is a part of a desktop computer,a laptop computer, etc., as described in further detail hereinbelow.

Optionally, the computer controller 130 is a part of a device installedin a car, as described in further detail hereinbelow.

Optionally, the computer controller 130 is a part of a cellular phone(say a smart phone), as described in further detail herein below.

The computer controller 130 controls the computer device in accordancewith an interaction between the first image and the second image.Consequently, the computer controller 130 allows the user to control thecomputer device, by moving the hand for bringing the first image and thesecond image into the interaction, as described in further detailhereinbelow.

In one example, if the user moves his hand, the image presenter 120moves the first image over an image of a computer keyboard. If the usermoves his fingers in a typing movement, the computer controller 130controls the computer device, making the computer device respond as ifthe user types using a real keyboard, as described in further detailhereinbelow.

In a second example, if the user moves his hand, the image presenter 120moves the first image over an image of a computer mouse. If the usermoves his fingers in a clicking movement, the fingers of the first imagemove in a clicking movement accurately correlated with the movement ofthe hand's movement. Consequently, the computer device responds as ifthe user clicks on one of the buttons of a real computer mouse,according to position of the first image in relation to the image of thecomputer mouse (i.e. with the fingers of the first image positioned overthe button of the computer mouse in the image).

In a third example, the second image is a graphical user interface (GUI)of a computer application such as a Microsoft™ Excel spreadsheet withbuttons and menu options, a web page with hyperlinks, buttons, etc.

The computer controller 130 controls the computer device according tointeraction between the first image and the second image. The computerdevice is controlled, as if the first image is the user's own hand, andthe second image is the GUI presented on a touch screen which allows theuser to interact with the GUI's menu, buttons, etc., by touching thetouch screen with his fingers.

Optionally, the hand tracker 110 uses bi-dimensional video data, fortracking the movement of the hand, say bi-dimensional video imagesstreamed from a camera connected to the hand tracker 110, as describedin further detail hereinbelow.

Optionally, the hand tracker 110 tracks the movement by detecting asegmentation (i.e. division) of a continuous feature in a light patternprojected onto the hand. The light pattern has the continuous feature ina first direction, and a noncontinuous (say periodic) feature in adirection substantially perpendicular to the first direction, asdescribed in further detail hereinbelow.

In one example, the light pattern includes several stripes arranged inparallel (or in near parallel) to each other.

The hand tracker 110 may identify in the light pattern, one or moreclusters of one or more stripe segments created by segmentation ofstripes by a digit of the hand (i.e. a finger or a thumb), as describedin further detail hereinbelow.

The hand tracker 110 tracks movement of the digit, by tracking thecluster of stripe segments created by segmentation of stripes by thedigit, or by tracking at least one of the cluster's segments.

Optionally, the hand tracker 110 further identifies in the lightpattern, one or more clusters of one or more stripe segments created bysegmentation of stripes by a palm of the hand. The hand tracker 110tracks movement of the hand, by tracking the cluster of stripe segmentscreated by segmentation of stripes by the palm, or by tracking at leastone of the cluster's segments.

Optionally, the hand tracker 110 may further detect a shift in positionof a notation along a continuous feature in the light pattern projectedonto the hand. The hand tracker 110 uses the detected shift, in thetracking of the hand, as described in further detail hereinbelow.

Optionally, the hand tracker 110 further identifies a depth movement ofthe digit (say a movement similar to clicking or a touch screen likeoperation), as described in further detail hereinbelow.

Optionally, after the clusters of stripe segments created by the hand'sdigits and palms are detected, only the stripe segments of the palm anddigit clusters are tracked, thereby tracking the movement of the hands.Parts of video data excluding the stripe segment clusters, such asbackground stripe segments and additional image information may thus bedismissed.

Consequently, computational effort and processing time for tracking themovement may be significantly reduced.

Optionally, the apparatus 1000 further includes a light projector and acamera, in communication with the hand tracker 110, as described infurther detail hereinbelow.

In one example, the light projector, the camera, or both, are remotefrom the computer device, as described in further detail hereinbelow.

Optionally, the light projector includes a source of light and a microstructured element, as known in the art.

The source of light may a laser diode, a light emitting diode (LED), anyanother element which emits a light beam, as known in the art.

The light beam emitted by the light source propagates through the microstructured element, onto the user's hand. The micro structured elementmodifies the light beam, for generating the light pattern projected ontothe use's hand, as described in further detail hereinbelow.

Optionally, the micro structured element converts the light beam into alight beam of a variable cross sectional intensity profile.Consequently, the cross sectional intensity profile varies along thelight beam, thus providing information indicative of a distance of anobject (say one of the user's digits) from the source of light, asdescribed in further detail hereinbelow.

Optionally, the light projector projects a light pattern having acontinuous feature in a first direction and a non-continuous feature ina direction substantially perpendicular to the first direction, onto thehand, as described in further detail hereinbelow.

Optionally, the micro structured element is a diffractive opticalelement, as known in the art.

The diffractive optical element may be obtained by a periodicmicrostructure that splits the light beam into a discrete number ofspots having a one dimensional or a two dimensional spatial arrangement,as know in the art.

Optionally, an additional element such as a cylindrical micro lens arrayor an additional diffractive element is used to create a stripe fromeach of the spots.

Optionally, the stripe is marked with phase notations positioned infixed distances from each other, along the stripe, as described infurther detail hereinbelow.

A possible advantage of using an additional microstructure element suchas a cylindrical micro lens array is that the additional element maydisperse a zero order of a light beam such as a laser beam, as known inthe art.

The zero order is a center of a light spot output from the diffractiveelement, and may be characterized by relatively high energy. Dispersingthe zero order to a light structure such as the stripe, may allowincreasing intensity of a laser light beam without crossing eye safetylimits.

Optionally, the light projector further includes an additionaldiffractive element, which changes the cross-sectional intensity profileof each stripe in the light pattern with respect to the distance from anobject which reflects the projected stripe. The cross-sectionalintensity profile is an intensity profile perpendicular to thepropagation direction of the light projected.

Optionally, the intensity profile change is a gradual change from aGaussian cross-sectional intensity profile to a Top Hat cross-sectionalintensity profile, carried out gradually along a distance traversed bythe projected light, as the projected light propagates to an object, andis reflected back from surface of the object.

Optionally, the intensity profile change is a gradual change from anintensity profile with a single peak to an intensity profile with two ormore peaks, etc., as known in the art.

The change in the intensity profile, along distance traversed by lightreflected from an object projected by the light with the light pattern,may help differentiate between stripes reflected from different objects,in different ranges, and thus further help overcome a segmentunification problem, as discussed in further detail hereinbelow.

Further, the change in the intensity profile may further be used todirectly measure distance to the object (say to the user's hand, fingersand thumb).

The camera may be a video camera such as a webcam or a cellular phone'scamera, positioned next to the light projector.

The camera captures one or more image of the hand projected with thelight pattern, and forwards the captured images to the hand tracker 110.The hand tracker 110 uses the captured images for tracking the movementof the hand, as described in further detail hereinbelow.

Optionally, the apparatus 1000 further includes a gesture recognizer, incommunication with the hand tracker 110.

The gesture recognizer detects in the tracked movement of the hand, agesture predefined by an administrator of the apparatus 1000. Upon thedetection of the gesture, the computer controller 130 controls thecomputer device in a manner predefined for the detected gesture, asdescribed in further detail hereinbelow.

Optionally, upon detection of a specific predefined gesture, the imagepresenter 120 aligns the first image into a predefined position. In oneexample, if the user shakes his hand, the image presenter 120 aligns thefirst image into a central position over the second image (say thecomputer keyboard image).

Optionally, upon detection of the specific predefined gesture, the imagepresenter 120 resizes the first image (i.e. hand image), as described infurther detail hereinbelow.

Reference is now made to FIG. 2, which is a flowchart illustrating amethod for remote hand control of a computer device, according to anexemplary embodiment of the present invention.

An exemplary method according to an exemplary embodiment of the presentinvention may be implemented on the computer device, be the computerdevice a desktop computer, a laptop computer, a cellular phone, etc.

The exemplary method may also be implemented on a device connectable tothe computer device, say on a unit with a computer processor incommunication with the computer device, as described in further detailhereinabove.

In the exemplary method, there is tracked 210 movement of a hand of auser of the computer device, say using images of the hand projected withlight structured in a pattern useful for tracking fine movement offingers and thumbs, as described in further detail hereinbelow.Optionally, the movement is tracked using the hand tracker 110, asdescribed in further detail hereinabove.

Simultaneously, there are presented 220 a first image representative ofthe hand and a second image to the user, say on screen of the computerdevice. Optionally, the second image and the first image are presentedto the user by the image presenter 120, as described in further detailhereinabove.

Optionally, the second image is a graphical user interface (GUI) of acomputer application such as a Microsoft™ Excel spreadsheet with buttonsand menu options, a web page with hyperlinks, buttons, etc.

Optionally, the second image is rather representative of a computerinput device, say a keyboard or a computer mouse, as described infurther detail hereinbelow.

Optionally, the second image is another graphical object, a picture,etc. The first image may be a full and accurately animated image of theuser's hand, presented on the screen. Alternatively, the first imagerather consists of five cursors, which represent the locations of thehand's fingertips. The first image may also have any other visualappearance which gives the user information about the location of digits(i.e. fingers and thumb) in relation to the second image.

The first image is moved in concert with the user hand movement tracked210 by the hand tracker 110, as described in further detail hereinbelow.

The computer device is controlled 230, say by the computer controller130, in accordance with an interaction between the first image and thesecond image.

Consequently, the user is allowed to control the computer device, bymoving the hand for bringing the first image and the second image intothe interaction, as described in further detail hereinbelow.

In one example, if the user moves his hand, the first image is movedover an image of a computer keyboard.

If the user moves his fingers in a typing movement, the computer deviceis controlled 230, to make the computer device respond as if the usertypes using a real keyboard wired to the computer device, as describedin further detail hereinbelow.

In a second example, if the user moves his hand, the first image ismoved over an image of a computer mouse.

If the user moves his fingers in a clicking movement, the fingers of thefirst image move in a clicking movement accurately correlated with themovement of the hand's movement. Consequently, the computer deviceresponds as if the user clicks on a button of a real computer mouse,according to position of the first image in relation to the image of thecomputer mouse (i.e. with the fingers of the first image positioned overthe button of the computer mouse in the second image).

In a third example, the second image is a graphical user interface (GUI)of a computer application such as a Microsoft™ Excel spreadsheet withbuttons and menu options, a web page with hyperlinks, buttons, etc.

The computer device is controlled 230 (say by the computer controller130), according to interaction between the images. The computer deviceis controlled 230, as if the first image is the user's own hand, and thesecond image is a GUI presented on a touch screen which allows the userto interact with the GUI's menu, buttons, etc., by touching the touchscreen with his fingers.

Optionally, the tracking 210 of the movement of the hand is carried outusing bi-dimensional video data (say bi-dimensional video imagesstreamed from a camera connected to the hand tracker 110), as describedin further detail hereinabove.

Optionally, the movement is tracked 210, by detecting a segmentation(i.e. division) of a continuous feature in a light pattern projectedonto the hand. The light pattern has the continuous feature in a firstdirection, and a non-continuous (say periodic) feature in a directionsubstantially perpendicular to the first direction, as described infurther detail hereinbelow.

In one example, the light pattern includes several stripes arranged inparallel (or in near parallel) to each other.

Optionally, there may be identified in the light pattern, one or moreclusters of one or more stripe segments created by segmentation ofstripes by a digit of the hand (i.e. a finger or a thumb), as describedin further detail hereinbelow. Consequently, the movement of the digitis tracked 210, by tracking the cluster of stripe segments created bysegmentation of stripes by the digit, or by tracking at least one of thecluster's segments.

Optionally, there is further identified a depth movement of the digit(say a movement similar to a clicking or a touch screen like operation),say by detecting a change in the number of segments in the trackedcluster, as described in further detail hereinbelow.

Optionally, there are further identified in the light pattern, one ormore clusters of one or more stripe segments created by segmentation ofstripes by a palm of the hand. Consequently, the movement of the hand istracked 210, by tracking the cluster of stripe segments created bysegmentation of stripes by the palm, or by tracking at least one of thecluster's segments.

Optionally, there is further detected a shift in position of a notationalong a continuous feature in the light pattern projected onto the hand.The detected shift is used in the tracking 210 of the hand, say by thehand tracker 110, as described in further detail hereinbelow.

Optionally, the exemplary method further includes detecting a gesturepredefined by an administrator of the apparatus 1000, in the tracked 210movement of the hand. Upon the detection of the gesture, the computerdevice is controlled 230 in a manner predefined for the detected gesture(say by the computer controller 130), as described in further detailhereinbelow.

Optionally, upon detection of the predefined gesture, the first image isaligned into a predefined position. In one example, if the user shakeshis hand, the image presenter 120 aligns the first image into a centralposition over the second image (say the computer keyboard image or theGUI).

Optionally, upon detection of the predefined gesture, the first image isresized (say by the image presenter 120), as described in further detailhereinbelow.

Reference is now made to FIG. 3, which is a block diagram schematicallyillustrating a computer readable medium storing computer executableinstructions for performing steps of remote hand control of a computerdevice, according to an exemplary embodiment of the present invention.

According to an exemplary embodiment of the present invention, there isprovided a computer readable medium 3000, such as a CD-ROM, aUSB-Memory, a Portable Hard Disk, a diskette, etc.

The computer readable medium 3000 stores computer executableinstructions, for performing steps of remote hand control of a computerdevice, according to an exemplary embodiment of the present invention.

The computer executable instructions include a step of tracking 310movement of a hand of a user of the computer device, say using images ofthe hand projected with light structured in a pattern useful fortracking fine movement of fingers and thumbs, as described in furtherdetail hereinbelow.

The computer executable instructions further include a step ofpresenting 320 a first image representative of the hand and a secondimage, to the user, say on screen of the computer device, as describedin further detail hereinabove.

In the step 320, the first image is further moved in concert with theuser hand movement tracked 310, as described in further detailhereinbelow.

Optionally, the second image is a graphical user interface (GUI) of acomputer application such as a Microsoft™ Excel spreadsheet with buttonsand menu options, a web page with hyperlinks, buttons, etc.

Optionally, the second image is rather representative of a computerinput device, say a keyboard or a computer mouse, as described infurther detail hereinbelow.

Optionally, the second image is another graphical object, a picture,etc. The first image may be a full and accurately animated image of theuser's hand, presented on a screen. Alternatively, the first imagerather consists of five cursors, which represent the locations of thehand's fingertips. The first image may also have any other visualappearance which gives the user information about the location of digits(i.e. fingers, thumb, or both) in relation to the second image.

The computer executable instructions further include a step in which thecomputer device is controlled 330, in accordance with an interactionbetween the first image and the second image.

Consequently, the user is allowed to control the computer device, bymoving the hand for bringing the first image and the second image intothe interaction, as described in further detail hereinbelow.

In one example, if the user moves his hand, the first image is movedover an image of a computer keyboard.

If the user moves his fingers in a typing movement, the computer deviceis controlled 330, to make the computer device respond as if the usertypes using a real keyboard, as described in further detail hereinbelow.

In a second example, if the user moves his hand, the first image ismoved over an image of a computer mouse.

If the user moves his fingers in a clicking movement, the fingers of thefirst image move in a clicking movement accurately correlated with themovement of the hand's movement. Consequently, the computer deviceresponds as if the user clicks on one of the buttons of a real computermouse, according to position of the first image in relation to the imageof the computer mouse (i.e. with the fingers of the first imagepositioned over a button of the mouse in the second image).

In a third example, the second image is a graphical user interface (GUI)of a computer application such as a Microsoft™ Excel spreadsheet withbuttons and menu options, a web page with hyperlinks, buttons, etc.

The computer device is controlled according to interaction between theimages, as if the first image is the user's own hand, and the secondimage is a GUI presented on a touch screen which allows the user tointeract with the GUI's menu, buttons, etc., by touching the touchscreen with his fingers.

Optionally, the tracking 310 of the movement of the hand is carried outusing bi-dimensional video data (say bi-dimensional video imagesstreamed from a camera), as described in further detail hereinabove.

Optionally, the movement is tracked 310, by detecting a segmentation(i.e. division) of a continuous feature in a light pattern projectedonto the hand. The light pattern has the continuous feature in a firstdirection, and a non-continuous feature in a direction substantiallyperpendicular to the first direction, as described in further detailhereinbelow.

In one example, the light pattern includes several stripes arranged inparallel (or in near parallel) to each other.

Optionally, the instructions further include identifying in the lightpattern, one or more clusters of one or more stripe segments created bysegmentation of stripes by a digit of the hand (i.e. a finger or athumb), as described in further detail hereinbelow. Consequently, themovement of the digit may be tracked 310, by tracking the cluster ofstripe segments created by segmentation of stripes by the digit, or bytracking at least one of the cluster's segments.

Optionally, the instructions further include identifying a depthmovement of the digit, say by detecting a change in the number ofsegments in the tracked cluster, as described in further detailhereinbelow.

Optionally, the instructions further include identifying in the lightpattern, one or more clusters of one of more stripe segments created bysegmentation of stripes by a palm of the hand. Consequently, themovement of the hand may be tracked 310, by tracking the cluster ofstripe segments created by segmentation (i.e. division) of stripes bythe palm, or by tracking at least one of the cluster's segments.

Optionally, the instructions further include detecting a shift inposition of a notation along a continuous feature in the light patternprojected onto the hand. The detected shift is used for tracking 310 ofthe hand, as described in further detail hereinbelow.

Optionally, the instructions further include detecting a gesturepredefined by an administrator, in the tracked 310 movement of the hand.Upon the detection of the gesture, the computer device is controlled 330in a manner predefined for the detected gesture, as described in furtherdetail hereinbelow.

Optionally, the instructions further include a step in which, upondetection of the predefined gesture, the first image is aligned into apredefined position. In one example, if the user shakes his hand, thefirst image is aligned into a central position over the second image(say the computer keyboard image or GUI).

Optionally, the instructions further include a step in which, upondetection of the predefined gesture, the first image is resized, asdescribed in further detail hereinbelow.

Reference is now made to FIG. 4, which schematically illustrates a handprojected with a light pattern, according to an exemplary embodiment ofthe present invention.

According to an exemplary embodiment, the tracking of the movement ofthe hand is carried out, using a light pattern designed to enabledetection of hand movement, such as fine movements of fingers andthumbs.

The specifically designed light pattern allows the tracking of themovement, even in bi-dimensional video data, which unlike threedimensional depth map, does not provide for easy separation of the handsfrom the rest of the body, according to distance, as described infurther detail hereinabove.

Optionally, the light pattern is specifically designed to track movementof the hand's digits in a bi-dimensional video data (say video imagesstreamed from a regular video camera). More specifically, the lightpattern may be designed to enable detection and tracking of digits (i.e.fingers and thumb) as well as palm, in the bidimensional video data,according to distortions of the pattern by the digits, as described infurther detail hereinbelow.

Optionally, the light pattern has a continuous feature in a firstdirection (say the X-axis) and a non-continuous (say periodic) featurein a direction substantially perpendicular to the first direction (saythe Y-axis).

In one example for such a pattern, the light pattern includes severalstripes arranged in parallel (or in near parallel) to each other, asillustrated schematically using FIG. 4.

A camera is positioned in a certain Y-axis distance, above a lightprojector which projects the stripes pattern on the hand 410 and on thebackground 420 (say a surface of a table the hand rests on, a wall,etc.).

The position of the camera is selected, so as to create a triangulationeffect between the camera, the light projector and the light reflectedback from the user's hand 410 and the background 420, as known in theart.

The triangulation effect causes discontinuities in the pattern at thepoints along a stripe where there are significant depth shifts from anobject projected with a light pattern.

The discontinuities segment (i.e. divide) the stripe into two or morestripe segments, say a segment 431 positioned on the hand, a segment 432position to the left of the hand and a segment 433 position to the rightof the hand.

Such depth shift generated stripe segments may be located on thecontours of the user's hand's palm or digits, which are positionedbetween the camera and the user's body.

That is to say that the user's digit or palm segments the stripe intotwo or more stripe segments.

Once such a stripe segment is detected, it is easy to follow the stripesegment, to the stripe segment's ends.

The hand tracker 110 may thus analyze bi-dimensional video data (say avideo stream forwarded to the hand tracker 110 from the camera), togenerate clusters of stripe segments.

For example, the hand tracker 110 may identify in the light pattern, acluster of one or more stripe segments created by segmentation ofstripes by a digit of the hand, say a cluster 441 of four segmentsreflected from the hand's central finger. Consequently, the hand tracker110 tracks the movement of the digit, by tracking the cluster of stripesegments created by segmentation of stripes by the digit, or by trackingat least one of the cluster's segments.

The cluster of stripe segments created by segmentation (i.e. division)of stripes by the digit includes strip segments with an overlap in the Xaxis. Optionally, the stripe segments in the cluster further havesimilar lengths (derived from the fingers thickness) or relativeproximity in the Y-axis coordinates.

On the X-axis, the segments may have a full overlap for a digitpositioned straightly, or a partial overlap for a digit positioneddiagonally in the X-Y plane.

Optionally, the hand tracker 110 further identifies a depth movement ofthe digit, say by detecting a change in the number of segments in thetracked cluster.

For example, if the user stretches the user's central digit, the anglebetween the digit and the plane of the light projector and camera (X-Yplane) changes. Consequently, the number of segments in the cluster 441is reduced from four to three.

Optionally, the hand tracker 110 further identifies in the lightpattern, one or more clusters of one or more stripe segments created bysegmentation of stripes by a palm of the hand. Consequently, themovement of the hand is tracked 210, by tracking the cluster of stripesegments created by segmentation of stripes by the palm, or by trackingat least one of the cluster's segments.

The cluster of stripe segments created by segmentation of stripes by thepalm includes an upper strip segment 431 which overlaps with the userhand's fingers stripe segment clusters, in the X axis.

The upper strip segment 431 overlaps the four finger clusters in theX-axis, but do not exceed beyond the minimum and maximum X value of thefour finger clusters' bottom segments.

The cluster of stripe segments created by segmentation of stripes by thepalm further includes, just below segment 431, a few strip segments insignificant overlap with the strip segment 431. The cluster of stripesegments created by segmentation of stripes by the palm further includeslonger stripe segments that extend to the base of a stripe segmentcluster 451 of the user's thumb.

The digit and palm cluster's orientation may differ with specific handspositions and rotation.

Reference is now made to FIG. 5, which schematically illustrates a handprojected with a light pattern having an intensity notation, accordingto an exemplary embodiment of the present invention.

One problem with the light pattern illustrated hereinabove is a stripesegment unification problem.

In certain cases, stripe segments reflected from background of the handmay unite with stripe segments reflected from the hand. Consequently,the stripe segment clusters of the palm and digits cannot be found, andthe hands cannot be tracked.

One way that may help overcome the stripe segment unification problem,is through introduction of a periodic notation along each stripe, asillustrated in FIG. 5.

Optionally, the notation is in a form of a light intensity 320, whichchanges along the stripe, or a specific shape that appears in a constantdistance along the stripe, such as a short vertical line.

Further to the introduction of the periodic notation, the camera ispositioned with respect to the light projector, with a gap in theX-axis, as well as the gap in the Y-axis discussed in further detailhereinabove. This gap in the X-axis creates a phase shift in theposition of the notations along the stripes with respect to the object'sdistance.

Consequently, in cases in which the stripe segments reflected from thehands unite with strip segments reflected from the background, thesegments still have different phases of the periodic notation.Optionally, the ratio of the vertical and horizontal distances betweenthe camera and the light projector is carefully selected, so as tomaximize the notation's phase shift between the segments when thesegments unite.

A second way which may prove useful for overcoming the problem of theunited segments is by changing the setup of the camera and lightprojector, so as to differ in their Z-axis positions too.

A difference in Z-axis positioning between the camera and the lightprojector may be brought about either by physically positioning thecamera and light projector in different Z-axis positions or bydeployment of a negative or positive lens in front of the projectedpattern at a certain distance.

Consequently, the stripe segments of the light pattern reflected fromthe object have different periods at different object distances.Therefore, even if one of the stripe segments reflected from the handunites with a stripe segment reflected from the background, adjacentlines do not unite because of the different periods.

A third way which may prove useful for overcoming the problem of theunited segments, is carried out using a micro structured element, whichconverts the light beam into a light beam of a varying cross sectionalintensity profile, as described in further detail hereinabove.

The cross sectional intensity profile changes according to distance ofan object which reflects the light pattern, from the light projector.

Consequently, the cross sectional intensity profile of the stripesegments reflected from the hand and the cross sectional intensityprofile of the stripe segments reflected from the background, aredifferent, as described in further detail hereinabove.

A method according to an exemplary embodiment of the present inventionmay further include tracking of depth movement of the hand's digit, bydetecting a vertical movement of the cluster of stripe segments createdby segmentation of stripes by the digit, a horizontal movement ofnotations on the stripe segments, or both.

Optionally, the tracking of the depth movement of the digit is carriedout by detecting a vertical movement of the uppermost segment in thecluster of stripe segments created by segmentation of stripes by thedigit. The uppermost segment represents the tip of the digit, whichperforms the digit's most significant movement in the Z-Y plane.

Optionally, the tracking of the depth movement is carried out bydetecting a change in number of segments in the tracked cluster. That isto that a depth movement of the hand finger (i.e. movement in theZ-axis) may cause one of the tracked stripe segments of the digit tomove upwards and disappear, cause a new stripe segment to appear, etc.

Reference is now made to FIG. 6, which is a block diagram schematicallyillustrating remote hand control of a computer device, using an image ofa computer keyboard, according to an exemplary embodiment of the presentinvention.

According to an exemplary embodiment of the present invention, movementof a hand 610 of a user of a computer device (say a desktop computer) iscontinuously tracked.

The user is presented a first image 620 representative of the hand and asecond image representative of a computer input device, say a computerkeyboard, a computer mouse, a joystick, etc., as known in the art, on ascreen.

The first image 620 may be a full and accurately animated image of theuser's hand, presented on a screen. Alternatively, the first image 620may consist of five cursors, which represent the locations of the hand's610 fingertips. The first image 620 may also have any other visualappearance which gives the user information about the location of digits(i.e. fingers and thumb) in relation to a computer input device alsodisplayed on the screen.

As the user moves his hand 610 (or only his fingers or thumb), the firstimage 620 is moved in concert with the tracked movement. That is to saythat the movement of the first image 620 is correlated with the movementof the user's own hand 610. If the user moves his hand 610 to the left,the first image 620 moves to the left. If the user flexes one of hisfingers, the first image 620 also flexes the finger, etc.

The computer device is controlled according to interaction between thefirst image 620 and the second image, as if the hand in the image 620 isthe user's own hand 610, and the input device is a real computer inputdevice connected to the computer device. The computer input device mayinclude, but is not limited to a standard peripheral computer device,such as a keyboard, a joystick, a computer mouse, etc.

The user is thus allowed to remotely control the computer device, bymoving the user's hand 610, for bringing the first image 620 and thesecond image into the interaction.

Consequently, the remote control of the computer device, by movement ofthe user's hand 610, is based on a continuous user interface, ratherthan on an interface limited to a predefined set of specific gestures(though specific gestures may also be defined).

In one example, if the user moves his hand 610, the first image 620moves over an image of a computer keyboard, and if the user moves hisfingers in a typing movement, the computer device responds as if theuser types using a real keyboard, as described in further detailhereinbelow.

In one example the second finger of the first image 620 is just abovethe ‘C’ character on an image of a computer keyboard. When the usermakes a pressing movement with his hand's 610 second finger, the secondfinger of the first image 620 moves in parallel and presses the ‘C’character of the keyboard image presented on the screen. Consequently,the computer device responds as if the ‘C’ character of a real keyboardwired to the computer device is pressed by the user, using the user'sown finger.

In the example, the absolute location of the user hands in a threedimensional space is not relevant, and only the movement of the user'sfingers needs to be tracked.

The movement of the first image 620 though accurately correlated withthe movement of the user's own hand 610, may be carried out in adifferent scale. That is to say that the size of the first image 620 maydiffer from the size of the user's real hand 610.

Optionally, the gesture recognizer of apparatus 1000 further allows theuser the resize the first image 620, as described in further detailhereinbelow.

Reference is now made to FIG. 7, which is a block diagram schematicallyillustrating a first gesture for remote hand control of a computerdevice, according to an exemplary embodiment of the present invention.

An apparatus according to an exemplary embodiment of the presentinvention may include a gesture recognizer, as described in furtherdetail hereinabove.

A first exemplary gesture, according to an exemplary embodiment of thepresent invention, is a reset gesture in which the user shakes his hand710 in front of the camera (say by quick minor vibration of the fingersand hand in a lateral movement).

Upon detection of the reset gesture, the first image presented on thescreen, moves to a predefined position over the second image, say to apredefined position over a computer input device presented in the secondimage or to a position over a first line of a spreadsheet like graphicaluser interface presented on the screen.

In one example, upon detection of the exemplary reset gesture, the imagepresenter 120 moves the first image into a position over a displayedcomputer keyboard's center, as described in further detail hereinabove.

The exemplary reset gesture may further ease detection of a digitlocation, since the exemplary reset gesture comprises separate movementof each digit.

Reference is now made to FIG. 8, which is a block diagram schematicallyillustrating a second gesture for remote hand control of a computerdevice, according to an exemplary embodiment of the present invention.

A second exemplary gesture is a resize gesture, say a gesture in whichthe user turns is hand 810 in a predefined direction.

Optionally, upon detection of the exemplary resize gesture, the imagepresenter 120 changes the size of the first image displayed on thescreen, without changing the second image in terms of size or zoomlevel.

For example, a user who interacts with an image of a keyboard displayedin a full screen mode, may want to use a larger first image (say handimage), in order to get from one side of the keyboard to the other witha relatively small movement of his real hands (since the scale ofmovement depends on size of the hand image). However, the user may needa smaller first image (say hand image), for selecting a text with asmall font on his web browser.

Alternatively, the size of the first image (i.e. hand image) remainsconstant, and the image presenter 120 rather resizes the second image(say the keyboard), which shrinks, expands, zooms in or zooms out, thuschanging the size proportion between the first image and the secondimage.

Reference is now made to FIG. 9, which is a block diagram schematicallyillustrating a third gesture for remote hand control of a computerdevice, according to an exemplary embodiment of the present invention.

A third exemplary gesture is a click (or a double click) gesture, say agesture in which the user clicks 910 using one or two fingers of theuser's hand.

Optionally, upon detection of the exemplary click (or a double click)gesture, the image presenter 120 moves fingers of the first image in aclicking movement accurately coordinated with the movement of the user'sfingers.

For example, the user may be presented an image of a computer mouse, andbe allowed to move the first image into a position above the computermouse in the image.

When the user moves his own fingers in a clicking movement, the fingersof the first image click on the image of the computer mouse, and thecomputer controller 130, makes the computer device respond as if theuser clicks on a real computer mouse with the user's own fingers.

Reference is now made to FIG. 10, which is a block diagram schematicallyillustrating a fourth gesture for remote hand control of a computerdevice, according to an exemplary embodiment of the present invention.

A fourth exemplary gesture is a zooming gesture, such as a gesture inwhich the user zooms in or simply magnifies an image presented on screenof a computer device, say by moving the user's hand 1010 in a pullingmovement or by moving two of the user's fingers away from each other(similarly to movement of fingers away from each other on a touch screenof a smart phone).

In one example, upon detection of the movement of the fingers away fromeach other, the image presenter 120 moves two fingers of the first imageaway from each other in parallel to the movement of the user's fingers.The computer controller 130 controls the computer device (say a cellularsmart phone), for zooming in or magnifying the image presented on screenof the computer device, thus providing the user with a touch screen likefunctionality without having the user touch a real touch screen.

Exemplary embodiment of the present invention provide for a variety ofother user gestures, say a scrolling down gesture in which the userstraitens his finger, or a scrolling up gesture in which the user moveshis finger in the opposite direction. The gestured may be carried withone or two hands, one or more digits, hand and digits, etc.

However, with exemplary embodiment of the present invention, the remotecontrol of the computer device, by movement of the user's hand, is basedon a continuous user interface, rather than on an interface limited to apredefined set of specific gestures (though specific gestures may alsobe defined).

For example, the remote control of the computer device may be based on acontinuous interaction of the user with the image of the computerkeyboard presented on screen of the computer device. The user is allowedto move his hand in the air, thus moving the first image over thekeyboard, to move the user's digits in a typing movement, thus typing onthe keyboard with the fingers of the first image, etc., as described infurther detail hereinabove.

In another example, the moves his hand, for moving the first imagethrough an interaction, in which the first image holds a computer mousepresented on screen, drags the mouse, etc. The computer device moves acursor on screen, as if a real mouse is dragged by the user, using isown hand.

Optionally, in the example, the user is further allowed to cause themouse to move automatically, by a single small middle finger movement.Consequently, the mouse moves automatically from one side of the screento the other, or rather the mouse moves through a predefined distance ortime (say for three seconds).

In yet another example, a graphical user interface (GUI) of a computerapplication such as a Microsoft™ Excel spreadsheet with buttons and menuoptions, a web page with hyperlinks, buttons, etc., is presented to theuser on screen of the computer device.

The computer device is controlled according to a continuous interactionbetween a hand image moving in concert with movement of a user's ownhand and fingers, and the GUI's menu, buttons, etc., as if the usertouches a touch screen which presents the GUI to the user, with theuser's own fingers.

Reference is now made to FIG. 11, which is a block diagram schematicallyillustrating a system for remote hand control of a computer device,according to an exemplary embodiment of the present invention.

An exemplary system for remote hand control of a computer deviceincludes the parts of apparatus 1000, as described in further detailhereinabove.

Optionally, one or more of the parts of the apparatus 1000, say thecamera and the light projector are installed on a mobile unit 1110. Themobile unit 1110 wirelessly communicates with other parts of theapparatus 1000, with the computer device 1120 (say a computer combinedwith a screen, like an Apple™ iMac computer, a TV set with a computerprocessor, etc.), or both.

The user may place the mobile unit 1110 on a table.

Then, the user may sit on a chair next to the table, move his hands, andthereby control the computer device 1120. The user controls the computerdevice 1120, through an interaction of the first image (representativeof the user's hand) with an image of a computer keyboard presented on ascreen of the computer device 1120, as described in further detailhereinabove.

The exemplary method and apparatus of the present invention may be usedfor additional applications.

For example, on a cellular phone, apparatus 1000 may enable the user toscroll, type text, zoom and click, without touching the cellular phone.Consequently, the cellular phone may be small, but may still allowinteraction with the cellular phone, as if the phone has a full sizecomputer keyboard.

Optionally, the cellular phone has a projector, which may project imageson a wall. Using the apparatus 1000, the user may move a first imageprojected next to an image of the phone's controller, be the controllera regular phone button or a touch screen.

Consequently, the user may bring the first image (say hand image) intointeraction with the image of the phone's controller, thus operating thephone in accordance with the interaction, as if the user uses his ownhand to operate the real phone controller directly.

In another example, the computer device is a computer device installedin a car (say a GPS unit or a phone unit).

Operating the computer device while driving is dangerous by nature ofinterfaces typically provided by such devices (i.e. small buttons ortouch screen), as a driver has to look to the side, for operating thecomputer device, rather than to the road, through the front window ofthe car.

The apparatus 1000 of the present embodiments may be installed on thecar's computer device (or in communication with the car's computerdevice).

The hand tracker 110 may track hand and finger movement of the driver,and thereby enable the driver to control the computer device (say theGPS or phone unit), without moving the driver's eyes from the road, tohis fingers which operate the small buttons of the unit.

Optionally, an image of a hand and an image of the GPS or phone unit'skeys are presented to the driver on a small screen installed on the carsfront window, without significantly blocking sight through the window.

The actual and exact location of the driver's own hand may change.

That is to say that the driver may click wherever he finds convenience,to actually dial the numbers or choose his destination in a guidingsystem of the GPS unit.

Consequently, the driver may control the GPS or phone unit, using hishand on an arbitrary area of the drive wheel, while looking through thefront window on which the screen is installed, and keeping his eyes onthe road.

Further, the driver may shake his hands in a reset movement, asdescribed in further detail hereinabove. Consequently, the first imageis positioned on center of an image of the car phone's numbers or thecar's GPS unit touch screen buttons, present on the screen.

The apparatus 1000 may also be installed on a musical instrument, say asynthesizer, a DJ turn table, or another musical instrument guided byhand movements of the user, as described in further detail hereinabove.

It is expected that during the life of this patent many relevant devicesand systems will be developed and the scope of the terms herein,particularly of the terms “Computer”, “CD-ROM”, “USB-Memory”, “HardDisk”, “Camera”, “Diffractive Optical Element”, “Laser Diode” and “LED”,is intended to include all such new technologies a priori.

It is appreciated that certain features of the invention, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention, which are, for brevity, described in thecontext of a single embodiment, may also be provided separately or inany suitable subcombination.

Although the invention has been described m conjunction with specificembodiments thereof, it is evident that many alternatives, modificationsand variations will be apparent to those skilled in the art.Accordingly, it is intended to embrace all such alternatives,modifications and variations that fall within the spirit and broad scopeof the appended claims.

All publications, patents and patent applications mentioned in thisspecification are herein incorporated in their entirety by referenceinto the specification, to the same extent as if each individualpublication, patent or patent application was specifically andindividually indicated to be incorporated herein by reference. Inaddition, citation or identification of any reference in thisapplication shall not be construed as an admission that such referenceis available as prior art to the present invention.

Further General Discussion

Some methods include regular image analysis from a digital video stream.

The video images are therefore analyzed to detect a specific area ofinterest, usually a body part such as a hand, a face, an arm, etc.

After detecting the body part location and position, body gestures maybe detected and translated to control media displayed on a computerscreen or TV. Detecting a body part such as the user's hand and fingersrequires first to analyze each pixel in the image by its color valuesand to compare it to other pixels in its environment. Such a method maysuffer from high computational time and from relatively high percentageof errors and false readings. One of the main difficulties comes fromthe background behind the hands needed to be detected. For example, themethod may have difficulties detecting the hands of a user that wears ashirt with a color that is close to his skin color or with anothercomplex pattern. The lightning of the room affects also the detectionperformance. The angle of the hands towards the camera may be adifficulty in this method as well. The shape of the hand changes withthe rotation of it and the fingertips may be on the same axis with therest of the hand towards the camera, and therefore be not detectable bycolor difference.

Other methods for hand detection and gesture recognition include using a3D depth map. The depth map is an image that holds in each pixel thedistance to the part of the object located in front of the depth camera,which is represented by the pixel. A depth solves many of the problemsthat exist in regular 2D video analysis. Extracting the hands locationis relatively easy since they are usually located in front of the bodyand therefore can be separated by cutting out the parts of the imagethat exceed a certain distance. With such methods, the angle of thehands towards the camera is less significant in the same manner. Theclosest part of the hand or the body can be detected by just thedistance, without considering the background color. However, acquiring afull 3D depth map requires a relatively complex hardware andcomputational effort.

The full depth map of a human body may be significant for certainapplications of gesture recognition such as gaming. For example, in atennis game application it is important exactly how the user is located,if his hand is stretched back ready to hit the ball or stretched forwardto block the ball near the net. However, other applications such as agesture driven virtual keyboard or mouse may not have the need for full3D depth mapping of hands and fingers, in order to control theapplications.

Therefore, extraction of the gestures directly from a light patternimage without acquiring a full 3D depth map as an intermediate step, astaught in further detail hereinabove, may prove advantageous.

Image comparison based solutions may use structured light patterns suchas a grid pattern in order to directly detect gestures. The comparisonbased solutions detect distortion in the grid pattern and compare thedistortions to a pre defined set of distortions of the grid pattern thatrepresent specific gestures. Once detected, the gestures activatecomputer actions which are attached to each gesture. Although the imagecomparison based solutions extract hand gestures directly for a capturedlight pattern image, they are limited to a discrete set of predefinedgestures, and do not offer a flexible tracking and representation of theuser hands and fingers, and the controlling of a computer device througha continuous user interface, as described in further detail hereinabove.

1. A computer implemented method comprising: obtaining a time series ofimages of a hand of a user of a computer device, the hand projected witha light structured in a pattern that enables tracking the hand, thepattern of the structured light comprises a continuous feature in afirst direction and a non-continuous feature in a second direction;detecting segmentation of the continuous feature based on changes inintensity profiles of light reflected from the hand along the firstdirection; tracking a movement of the hand using the images byidentifying, in the pattern of the structured light, at least onecluster of continuous feature segments created by the segmentation ofthe continuous feature by at least one part of the hand and tracking theat least one cluster of continuous feature segments; presenting, on adisplay of the computer device, a first image representative of the handand a second image representative of an object of a computer applicationrunning on the computer device; moving the first image in concert withthe tracked movement of the hand; detecting an interaction between thefirst image and the second image, the interaction represented by thefirst image with respect to the second image on the display of thecomputer device; and invoking a functionality of the computerapplication based on the detected interaction.
 2. The method of claim 1,wherein the continuous feature in the first direction comprises aplurality of stripes arranged substantially in parallel to each other,and the second direction is perpendicular to the first direction.
 3. Themethod of claim 1, wherein the tracking of the movement of the handcomprises using bi-dimensional video data.
 4. The method of claim 1,wherein the pattern of the structured light enables tracking movement ofdigits and a palm of the hand in bi-dimensional video data based ondistortions of the pattern by the digits and the palm.
 5. The method ofclaim 1, wherein the tracking of the movement of the hand furthercomprises identifying a depth movement of the at least one part of thehand, by detecting a change in a number of the continuous featuresegments in the at least one tracked cluster.
 6. The method of claim 1,wherein the pattern of the structured light comprises a grid pattern,and the method further comprises: detecting a predefined gesture in themovement of the hand by projecting the structured light having the gridpattern onto the hand.
 7. An apparatus, comprising: a light projectorconfigured to project, onto a hand of a user of a computer device, alight structured in a pattern that enables tracking of the hand, thepattern of the structured light comprises a continuous feature in afirst direction and a non-continuous feature in a second direction; acamera configured to capture a time series of images of the hand bycapturing images of the hand projected with the pattern of thestructured light; a hand tracker, associated with the camera, configuredto detect segmentation of the continuous feature based on changes inintensity profiles of light reflected from the hand along the firstdirection, the hand tracker is further configured to track a movement ofthe hand using the images by identifying, in the pattern of thestructured light, at least one cluster of continuous feature segmentscreated by the segmentation of the continuous feature by at least onepart of the hand and tracking the at least one cluster of continuousfeature segments; an image presenter, associated with the hand tracker,configured to present, on a display of the computer device, a firstimage representative of the hand and a second image representative of anobject of a computer application running on the computer device; and acomputer controller, associated with the image presenter, configured tomove the first image in concert with the tracked movement of the hand,wherein the computer controller is further configured to detect aninteraction between the first image and the second image, theinteraction represented by the first image with respect to the secondimage on the display of the computer device, and invoke a functionalityof the computer application based on the detected interaction.
 8. Theapparatus of claim 7, wherein the continuous feature in the firstdirection comprises a plurality of stripes arranged substantially inparallel to each other, and the second direction is perpendicular to thefirst direction.
 9. The apparatus of claim 7, wherein the hand trackeris further configured to use bi-dimensional video data for tracking themovement of the hand.
 10. The apparatus of claim 7, wherein the patternof the structured light enables tracking movement of digits of the handin bi-dimensional video data.
 11. The apparatus of claim 7, wherein thepattern of the structured light enables tracking movement of digits anda palm of the hand in bi-dimensional video data based on distortions ofthe pattern by the digits and the palm.
 12. The apparatus of claim 7,wherein the hand tracker is further configured to identify a depthmovement of the at least one part of the hand, by detecting a change ina number of the continuous feature segments in the at least one trackedcluster.
 13. The apparatus of claim 7, wherein the camera is furtherconfigured to capture the first image by capturing an image of the handprojected with the pattern of the structured light.
 14. The apparatus ofclaim 7, wherein the light projector is remote from the computer device.15. The apparatus of claim 7, wherein the camera is remote from thecomputer device.
 16. The apparatus of claim 7, further comprising agesture recognizer, associated with the hand tracker, configured todetect a predefined gesture in the tracked movement of the hand byprojecting the structured light having a grid pattern onto the hand,wherein the computer controller is further configured to control thecomputer device in a manner predefined for the detected gesture.
 17. Theapparatus of claim 16, wherein the image presenter is further configuredto carry out one member of the group consisting of: aligning of thefirst image into a predefined position, upon the detection of thepredefined gesture, resizing the first image, upon the detection of thepredefined gesture, and resizing the second image, upon the detection ofthe predefined gesture.
 18. The apparatus of claim 7, wherein thecomputer controller is a part of one member of the group consisting of amobile phone, a device installed in a car and a musical instrument. 19.The apparatus of claim 7, wherein the hand tracker is remote from thecomputer device.
 20. A non-transitory computer readable medium storingcomputer executable instructions for performing steps, the stepscomprising: obtaining a time series of images of a hand of a user of acomputer device, the hand projected with a light structured in a patternthat enables tracking the hand, the pattern of the structured lightcomprises a continuous feature in a first direction and a non-continuousfeature in a second direction; detecting segmentation of the continuousfeature based on changes in intensity profiles of light reflected fromthe hand along the first direction; tracking a movement of the handusing the images by identifying, in the pattern of the structured light,at least one cluster of continuous feature segments created by thesegmentation of the continuous feature by at least one part of the handand tracking the at least one cluster of continuous feature segments;presenting, on a display of the computer device, a first imagerepresentative of the hand and a second image representative of anobject of a computer application running on the computer device; movingthe first image in concert with the tracked movement of the hand;detecting an interaction between the first image and the second image,the interaction represented by the first image with respect to thesecond image on the display of the computer device; and invoking afunctionality of the computer application based on the detectedinteraction.